<html>
<head>
<title>Lightest Tutorial</title>
<style type="text/css">
body {
    margin-left: 5%;
    margin-right: 5%;
}
dt {
    font-weight: bolder;
}
dd {
    margin-top: 10px;
    margin-bottom: 10px;
}
pre {
    margin: 10px;
    padding: 10px;
    background-color: #eaeaea;
}
code {
    padding: 2px;
    background-color: #dfd;
}
table {
    border-collapse: collapse;
    border: solid 1px black;
}
th, td {
    border: solid 1px grey;
    padding: 5px;
}
.highlight {
    background-color: #eea;
}
.deprecated {
    font-style: italic;
    color: #c99;
}
</style>
</head>

<body>

<h1>Lightest Tutorial</h1>

<h2>Introduction</h2>

<p>Lightest is a lightweight, task-oriented functional and integration testing framework. It is designed to jumpstart custom testing framework creation by removing common obstacles such as test syntax derivation, test execution management, and test result reporting. It stands on the shoulders of <a href="http://testng.org/">TestNG</a> and built with the capable goodness of <a href="http://groovy.codehaus.org/">Groovy</a>.</p>

<p>Lightest truly enables test automation on all scales, not just on the unit level. And it's so light, you'll forget it's even there.</p>

<p>This tutorial steps you through the process of installing Lightest, creating basic testing tasks and tests, and understanding the major benefits the framework has to offer. To get through this tutorial, you'll need to have working knowledge of Java and/or Groovy, or a similar language such as C#. Examples abound, so don't be afraid to dive right in!</p>

<h2>Getting Lightest</h2>

<p>You can get a copy of Lightest from the <a href="http://code.google.com/p/lightest/downloads/list">Google Code project download page</a>. The current version is <code>0.1</code> . The download archive contains the executable jar, license, and some documentation. Lightest is made available under the <a href="http://www.apache.org/licenses/LICENSE-2.0.html">Apache 2.0 license</a>.</p>

<p>Lightest requires a version of Java that supports annotations, and a recent version of Groovy. I'm currently running Java <code>1.5.0_16</code> and Groovy <code>1.5.6</code>; you should be fine if you have ballpark versions.</p>

<p>For this tutorial, simply unpack the distribution.</p>

<h2>Hello World!</h2>

<p>Let's start by creating a task which prints "Hello World!" to the console. Each task is written as its own Groovy class, and implements the <code>ITask</code> interface. Most of the work is done by the task's <code>perform()</code> method. To keep things nice and easy, we'll bootstrap our first task class by extending <code>LightestTask</code>, which adds some default behaviors for other <code>ITask</code> methods, and allows us to specify the task logic in a single <code>doPerform()</code> method.</p>

<em>HelloWorld.groovy</em>
<pre>
import com.googlecode.lightest.core.LightestTask
import com.googlecode.lightest.core.ITaskResult

class HelloWorld extends LightestTask {
    
    void doPerform(ITaskResult result) {
        println 'Hello World!'
    }
}
</pre>

<p>Let's go ahead an whip up a quick test that uses our shiny new task!</p>

<em>Tutorial.groovy</em>
<pre>
import com.googlecode.lightest.core.LightestTestCase
import com.googlecode.lightest.core.SimpleApi
import org.testng.annotations.*

class Tutorial extends LightestTestCase {

    Tutorial() {
        def api = new SimpleApi(this)
        setApi(api)
    }
    
    @Test
    void sayHello() {
        HelloWorld ()
    }
}
</pre>

<p>All Lightest tests extend <code>LightestTestCase</code>. Since they are also TestNG tests, they make use of TestNG annotations, such as <code>@Test</code>, which indicates that a given method should be considered a test method. To invoke the HelloWorld task, we simply provide the name of the task class - which is HelloWorld! Now let's actually give our greeting by running this test. There is one final piece of the puzzle before we can do that - the configuration file.</p>

<em>tutorial.config</em>
<pre>
config {
    classPaths {
        path ('/Users/chai/work/tutorial')
    }
}
</pre>

<p>For now, we're putting all of our Groovy files in the same directory. I'm putting them in <em>/Users/chai/work/tutorial</em> . I specify this path in the configuration file. You might notice that this configuration file takes on the Groovy builder syntax, which allows concise specification of nested data structures. You'll be seeing more builder syntax later.</p>

<p>We're ready for a test run. For simplicity's sake, I'll assume the Lightest core distribution jar is in the same folder along with the files we just created. You'll have to provide the path to the jar in your command line otherwise.</p>

<pre>java -jar lightest-core-0.1-standalone.jar tutorial.config Tutorial.groovy</pre>

<p>You should see some output like:</p>

<pre>
[Parser] Running:
  /private/tmp/lightest-suite33716.xml

Hello World!

===============================================
Suite1
Total tests run: 1, Failures: 0, Skips: 0
===============================================

</pre>

<p>And if you navigate to the <em>lightest-report</em> directory, you should see that some files were generated there. Open up <em>lightest-report/index.html</em> in a web browser, and you'll see an HTML report. Drill down into the suite, and mouse over the "HelloWorld" task. You should see it expand with details like this:</p>

<p style="text-align: center"><img src="tutorial-images/hello-world.png" /></p>

<p>Tasks are the fundamental building blocks of Lightest tests. While in any given test you are free to do just anything Groovy allows you to, tasks bring the coherence and reportability to your tests that has traditionally been lacking in pure unit testing frameworks. Lightest enable you to easily create the tasks you want - but of course, it is up to you to design tasks appropriate for your application or system under test. Lightest will help where it can!</p>

<p>Congratulations! You're a Lightest tester!</p>

<h2>Better Results, Now!</h2>

<p>Our HelloWorld task is great, but wouldn't it be even better if we could choose to give a slightly different greeting? After all, just saying "Hello World!" all the time might get a little boring. Let's parameterize the task! Modify it as follows (from here on in, <code>import</code> statements will be omitted unless they are significant):</p>

<em>HelloWorld.groovy</em>
<pre>
class HelloWorld extends LightestTask {
    
    void doPerform(ITaskResult result) {
        println <span class="highlight">config.'@greeting' ?:</span> 'Hello World!'
    }
}
</pre>

<p>If you run the task as you ran it before, the same result - "Hello World!" will be printed to the console. Modify your test to say a different greeting:</p>

<em>Tutorial.groovy</em>
<pre>
class Tutorial extends LightestTestCase {

    Tutorial() {
        def api = new SimpleApi(this)
        setApi(api)
    }
    
    @Test
    void sayHello() {
        HelloWorld ()
    }
    
<span class="highlight">    @Test
    void sayGreeting() {
        HelloWorld (greeting: "Top of the mornin', world!")
    }</span>
}
</pre>

<p>Running the test, we get output like:</p>

<pre>
[Parser] Running:
  /private/tmp/lightest-suite36602.xml

Top of the mornin', world!
Hello World!

===============================================
Suite1
Total tests run: 2, Failures: 0, Skips: 0
===============================================

</pre>

<p>Now there's no way anyone can catch us boring the world with our salutations! In our task, we made use of the <code>greeting</code> attribute of the <code>config</code> object to pass along information from the test. The <code>config</code> object is automatically available to any tasks that subclass <code>LightestTask</code>. All tasks defined in this tutorial will do so for convenience. The attribute is accessed using the Groovy <code>Node</code> attribute shorthand, namely the <code>@</code> prefix. The <code>?:</code> Groovy operator is very useful here; if the <code>greeting</code> attribute is <code>null</code>, we say "Hello World!" by default.</p>

<p>In the test, we can choose to specify the <code>greeting</code> value by using the Groovy <code>Map</code> syntax within the parentheses following the task call. Can't really get simpler than that.</p>

<p>Since nobody knows just <em>what</em> we might say at a given time, wouldn't it be nice if there were a record of what we said after the fact? After all, it could have been pithy genius - and we might not ever grace the world with the same greeting ever again. Let's transcribe our remarks to the generated report. To do this, we only need to modify the task.</p>

<em>HelloWorld.groovy</em>
<pre>
class HelloWorld extends LightestTask {
    
    void doPerform(ITaskResult result) {
        <span class="highlight">def greeting =</span> config.'@greeting' ?: 'Hello World!'
        
<span class="highlight">        println greeting
        result.setMessage("Said: ${greeting}")</span>
    }
}
</pre>

<p>We set the message on the <code>result</code> object passed into the task's <code>doPerform()</code> method to record what was actually said. Now if you take a look at the report, you'll find a few items of interest. First, there are now entries for both <code>sayHello()</code> and <code>sayGreeting()</code>. Separate TestNG test methods are reported independently. Second, the value that we used as the greeting in <code>sayGreeting()</code> actually <em>is</em> displayed, despite my indicating otherwise - in the parameter list next to the name of the task. The parameter list for our original test <code>sayHello()</code> is empty. However, if you mouse over the task, you will see that the message we printed is available there.</p>

<p>Next, we learn how to glean information about the context in which we're giving our salutations.</p>

<h2>Save The Environment</h2>

<p>Each test runs in an environment, which potentially contains information about settings and resources specific to the test. For example, a test that needs to access the filesystem may need to know the "home" folder of the application-under-test. So far, we haven't specified any environment data, and our tests have not depended on it. Let's add a new task that queries the environment for ... THE WORLD. For now, our task will simply be informational:</p>

<em>QueryWorld.groovy</em>
<pre>
class QueryWorld extends LightestTask {
    
    void doPerform(ITaskResult result) {
        result.setMessage("Current World: ${env.getWorld()}")
    }
}
</pre>

<p>The <code>env</code> object is another special object made available by <code>LightestTask</code>. It is set to the environment assigned to the running test. The environment itself has properties we will specify in a custom environment class that extends <code>ITestEnvironment</code>:</p>

<em>TutorialEnvironment.groovy</em>
<pre>
import com.googlecode.lightest.core.ITestEnvironment

class TutorialEnvironment implements ITestEnvironment {
    String id
    String world
}
</pre>

<p>This Groovy class simply declares the elements that can be configured in the environment. The only contract it must satisfy as an implementor of <code>ITestEnvironment</code> is that it must have <code>getId()</code> and <code>setId()</code> methods, which are automatically generated when you include the <code>id</code> property as shown above. We want the world to be specified in our environment, so we simply add another property called <code>world</code>.</p>

<p>Now we actually have to provide the environment information. To do this, we return to the configuration file:</p>

<em>tutorial.config</em>
<pre>
config {
    classPaths {
        path ('/Users/chai/work/tutorial')
    }
<span class="highlight">    envs (class: 'TutorialEnvironment') {
        env {
            id ('default')
            world ('Earth')
        }
    }</span>
}
</pre>

<p>We use the builder syntax again to declare <code>envs</code> as a list of environments that are of our type, <code>TutorialEnvironment</code>. Then we specify a "default" environment with its properties. With the environment in place, we should be able to use our newest task in a test:</p>

<em>Tutorial.groovy</em>
<pre>
class Tutorial extends LightestTestCase {

    Tutorial() {
        def api = new SimpleApi(this)
        setApi(api)
    }
    
    @Test
    void sayHello() {
        HelloWorld ()
    }
    
    @Test
    void sayGreeting() {
        <span class="highlight">QueryWorld ()</span>
        HelloWorld (greeting: "Top of the mornin', world!")
    }
</pre>

<p>Run the test, and take a look at the report. You should see an entry for the <code>QueryWorld</code> task, reporting a message of "Current World: Earth". No place like home!</p>

<p>A final note before moving on to the next topic: preferences are another configurable entity, like environments. However, while there may be multiple environments for a given test run, these is only one set of preferences across the run. The idea behind having multiple environments is that you may want to have your integration tests run in parallel across multiple enviroments, for faster feedback. However, the preferences are intended to be cross-cutting - for example a timeout interval. Just as our custom environment implemented the <code>ITestEnvironment</code> interface, custom preferences implement <code>IPreferences</code>, which is a marker interface only. Here's a peek at how you might specify preferences in your config file:</p>

<pre>
config {
    ...
    prefs (class: 'MyCustomPreferencesClass') {
        timeout (30)
        speed (100)
    }
    ...
}
</pre>

<h2>Turtles All The Way Down</h2>

<p>So far we've been executing single tasks, and in the last section, two tasks, in the <code>sayGreeting()</code> test. Let's explore one of the cooler features of Lightest - task nesting. Sometimes you might have a group of tasks related to a shared goal. If any of the tasks fail, the goal cannot be achieved. By the same token, if the tasks succeed, it would be convenient to represent them as a single lineitem rather than a series of distinct tasks.</p>

<p>We revisit the Groovy builder syntax. In Lightest tests, nested tasks can be specified in the same way that, for example, the configuration file specifies hierarchically organized values. With nested tasks, you have top level tasks, their children, and the children of their children - turtles all the way down. When all child tasks of a given parent task succeed (and the parent task itself is successful), the parent task is considered to have passed. If any child tasks fail, then the parent task is considered to have failed. If a parent task fails, the child tasks, which are assumed to have a dependency on the parent task, will not even be performed. We'll explore task failures further in the next section.</p>

<p>Greeting the world requires there to be a world. Let's modify our test a bit (from here on in, code samples will focus just on the test in question):</p>

<em>Tutorial.groovy</em>
<pre>
    @Test
    void sayGreeting() {
        QueryWorld () <span class="highlight">{</span>
            HelloWorld (greeting: "Top of the mornin', world!")
        <span class="highlight">}</span>
    }
</pre>

<p>Crack open the report and view the results for <code>sayGreeting()</code>. When you mouse over the <code>QueryWorld</code> task, it will expand. Notice one of the boxes in the upper-right corner of the task result, which will have one of the following values: &#19968;, &#20108;, or &#19977;. The default should be &#20108;. This refers to the display level of the task result, which you can toggle by clicking the header of the result - the line containing the task name and the task parameters. Click the header now. You should see the <code>HelloWorld</code> task result expand below the <code>QueryWorld</code> one, indented to indicate it is a child task. The task display level will now be &#19977;. Click it again to cycle to &#19968;, at which point the child result will hide itself again.</p>

<p>You can use various control constructs within Groovy builder closures, for example <code>if</code> statements and <code>for</code> loops. We could do something nifty like this, to showcase our mastery over greetings across regions and tongues:</p>

<em>Tutorial.groovy</em>
<pre>
<span class="highlight">    static final GREETINGS = [
        "Top of the mornin', world!"
        , "Howdy, world!"
        , "G'day mate, world!"
        , "Bonjour, world!"
    ]</span>
        
    @Test
    void sayGreeting() {
        QueryWorld (<span class="highlight">description: 'Find a world, and greet it internationally'</span>) {
            <span class="highlight">for (g in GREETINGS) {</span>
                HelloWorld (greeting: g)
            <span class="highlight">}</span>
        }
    }
</pre>

<p>When you run this test, you will see the following console output:</p>

<pre>
Top of the mornin', world!
Howdy, world!
G'day mate, world!
Bonjour, world!
</pre>

<p>And when you drill down into the <code>QueryWorld</code> task result, setting the task display level to &#19977; as before, you will find 4 <code>HelloWorld</code> child tasks. The important thing to note is that all of our tasks thus far has been successful. It is very nice that the child tasks roll up and are hidden with &#19968; and &#20108;. The special <code>description</code> value set to the top level task gives us enough of an idea of what the job of the entire "task tree" is, and is easily viewable in the <code>QueryWorld</code> task result. Since all tasks passed, the details are hidden from us by default, and we have little reason to worry about them.</p>

<p style="text-align: center"><img src="tutorial-images/international-hello.png" /></p>

<p>Another cool aspect of nested tasks is that they are able to obtain information from their parent tasks, and their associated results. In the following example, the <code>HelloWorld</code> task checks to see if it has a parent <code>QueryWorld</code> task, and if so adds the message from the parent result to the current result. The <code>?.</code> safe dereferencing Groovy operator is used heavily to avoid <code>NullPointerException</code>'s for tasks that do not have a parent.</p>

<em>HelloWorld.groovy</em>
<pre>
class HelloWorld extends LightestTask {
    
    void doPerform(ITaskResult result) {
        def greeting = config.'@greeting' ?: 'Hello World!'
<span class="highlight">        def parentResult = result.parent()
        
        if (parentResult?.getTask()?.class?.name == 'QueryWorld') {
            greeting += " (${parentResult.getMessage()})"
        }</span>
        
        println greeting
        result.setMessage("Said: ${greeting}")
    }
}
</pre>

<p>The message for the nested <code>HelloWorld</code> tasks should now be reported as something like:</p>

<pre>
Said: Howdy, world! (Current World: Earth)
</pre>

<p>While the current example seems fairly trivial (indeed, does it make sense to echo a message that is already available in the report?), it does illustrate that the result hierarchy can be easily navigated. This can be used to good effect when child tasks need to modify their behavior based on the configuration and/or result of their parent tasks.</p>

<p>So far things are going just peachy. We know how to write tasks, and provide information about the task run in the resulting report. We know how to get information about the environment, and to group related tasks via nesting. But now - you hear the ominous peal of thunder! In the next section, we'll experience stormier times...</p>

<h2>Abject _______</h2>

<p>In a way, tests are designed to fail. And the best tests give us lots of useful information when they fail. In this section, we explore task failures, especially failures in nested tasks. Let's have some fun!</p>

<em>WorldNotFoundException.groovy</em>
<pre>
class WorldNotFoundException extends Exception {
    WorldNotFoundException(String message) {
        super(message)
    }
}
</pre>

<p>Evil empires are known to say, "we'll blow your PLANET up!"</p>

<em>QueryWorld.groovy</em>
<pre>
class QueryWorld extends LightestTask {
    
    void doPerform(ITaskResult result) {
<span class="highlight">        def world = env.getWorld()
        
        assert world != null
        
        if ('Alderaan'.equalsIgnoreCase(env.getWorld())) {
            throw new WorldNotFoundException("World not found: ${world}")
        }
        </span>
        result.setMessage("Current World: ${env.getWorld()}")
    }
}
</pre>

<p>And the tweak to set the plan in motion:</p>

<em>tutorial.config</em>
<pre>
config {
    classPaths {
        path ('/Users/chai/work/tutorial')
    }
    envs (class: 'TutorialEnvironment') {
        env {
            id ('default')
            world ('<span class="highlight">Alderaan</span>')
        }
    }
}
</pre>

<p>Run the test! Now <em>that's</em> an unfamiliar sight in the console:</p>

<pre>
===============================================
Suite1
Total tests run: 2, <span class="highlight">Failures: 1</span>, Skips: 0
===============================================

</pre>

<p>Check the report. You'll find that our <code>sayGreeting()</code> test from the above section is highlighted in red. If you mouse over the <code>QueryWorld</code> task, you will see the message is "Unexpected exception: World not found: Alderaan", and the <code>WorldNotFoundException</code> stack trace is shown as the response data. And our nested <code>HelloWorld</code> task? It's nowhere to be found, even if you change the task display level. That's because when parent tasks fail, their child tasks are not even performed.</p>

<p>However, all is not lost. Unlike typical unit testing frameworks, the task-oriented nature of Lightest means the test may continue despite failures. See that in action by modifying the test to add a post-failure task:</p>

<em>Tutorial.groovy</em>
<pre>
    @Test
    void sayGreeting() {
        QueryWorld (description: 'Find a world, and greet it internationally') {
            for (g in GREETINGS) {
                HelloWorld (greeting: g)
            }
        }
        <span class="highlight">HelloWorld (greeting: 'Hello, non-blown-up world!')</span>
    }
</pre>

<p>The test is still considered a failure; however you can see that the non-failing <code>HelloWorld</code> task was still performed, and indicated success. This is because it does not have a parent-child relationship with the failing task.</p>

<p>What if the task that failed was the child task, not the parent? Let's add a new test called <code>sayHelloBeforeChecking()</code>:</p>

<em>Tutorial.groovy</em>
<pre>
    @Test
    void sayHelloBeforeChecking() {
        HelloWorld (greeting: 'Hello, non-blown-up world!') {
            QueryWorld (description: 'Oops, spoke too soon!')
            HelloWorld (greeting: 'Hello, Smith Areans!')
        }
    }
</pre>

<p>Ah ha, when a child task fails, the parent fails too - but sibling tasks are still executed and may succeed. In this case, you have to click on the parent <code>HelloWorld</code> task for <code>sayHelloBeforeChecking()</code> to get the passing child <code>HelloWorld</code> task to display. This is because of the "smart" hiding of passing child tasks discussed in the previous section.</p>

<p>We have just observed task failures due to exceptions being thrown by code within the task - the exception is caught and handled by the <code>LightestTask</code> superclass. Not all failures are caused by exceptions, however. Tasks (assertion tasks, for example) may check for conditions that, if unmet, will indicate the task has failed. They can do this by setting the status of the <code>result</code> object. Here we modify the <code>QueryWorld</code> task to <em>not</em> throw an exception:</p>

<em>QueryWorld.groovy</em>
<pre>
class QueryWorld extends LightestTask {
    
    void doPerform(ITaskResult result) {
        def world = env.getWorld()
        
        assert world != null
        
        if ('Alderaan'.equalsIgnoreCase(env.getWorld())) {
            <span class="highlight">//</span>throw new WorldNotFoundException("World not found: ${world}")
<span class="highlight">            result.setMessage("World not found: ${world}")
            result.fail()
        }
        else {</span>
            result.setMessage("Current World: ${env.getWorld()}")
<span class="highlight">        }</span>
    }
}
</pre>

<p>The task now fails without a stacktrace. It is key to indicate any available information about the failure using <code>setMessage()</code>, to aid in debugging.</p>

<p>The standard task result status when one is not set explicitly indicates success - that's why the earlier task implementations in this tutorial did not specify a status. Apart from <code>fail()</code>'ing the task, two other options are available to indicate anything but complete success: <code>flag()</code> and <code>doom()</code>. All three of these options result in failure for the containing test.</p>

<p><code>flag()</code> should be used to indicate a potential error condition when performing a task. The intent is for the troubleshooter to see the failed test, drill down into its tasks to see the flagged task, and determine whether or not there is a real issue. <code>doom()</code> should be used when little or no value will be gained from continuing to execute the test; all tasks that have not yet been performed will be abandoned. This is similar to the behavior of typical unit testing frameworks upon encountering an exception.</p>

<p>Experiment and have fun with the following example! Probably the most important thing to note is that any tasks following a <code>doom()</code>'ed task do not get executed.</p>

<em>QueryWorld.groovy</em>
<pre>
class QueryWorld extends LightestTask {
    
    void doPerform(ITaskResult result) {
        def world = env.getWorld()
        
        assert world != null
        
        if ('Alderaan'.equalsIgnoreCase(env.getWorld())) {
            //throw new WorldNotFoundException("World not found: ${world}")
            result.setMessage("World not found: ${world}")
            result.fail()
        }
<span class="highlight">        else if ('Atlantis'.equalsIgnoreCase(env.getWorld())) {
            result.setMessage("World quite difficult to find: ${world}")
            result.flag()
        }
        else if ('Hades'.equalsIgnoreCase(env.getWorld())) {
            result.setMessage("Unable to find more worlds: ${world}")
            result.doom()
        }</span>
        else {
            result.setMessage("Current World: ${env.getWorld()}")
        }
    }
}
</pre>

<p style="text-align: center"><img src="tutorial-images/doomed.png" /></p>

<p>TestNG is still hard at work behind the scenes, and this is evident when tests fail. In the reporting folder you will find a file called <em>testng-failed.xml</em>, which not only lists the failures as XML, but also allows you to easily rerun only the failed tests, with a command such as the following:</p>

<pre>
java -jar lightest-core-0.1-standalone.jar tutorial.config testng-failed.xml
</pre>

<p>Although you may specify multiple <em>.groovy</em> files to execute as tests when running Lightest, using TestNG's XML suite files gives much more flexibility. Consult the TestNG documentation for more information on the suite XML syntax.</p>

<h2>Taste of Paradise</h2>

<p>That just about wraps up our tour of Lightest. The Lightest project is really just getting underway, and there are exciting improvements on the horizon. To conclude, I'd like to share what I think some cool features would be:</p>

<ul>
<li>Grails webapp - test manager and execution engine.</li>
<li>Persisting results to a database, and mining the results for comparisons.</li>
<li>A Java annotation editor (again via a webapp), so the annotation-based metadata about tests can be managed with ease.</li>
<li>An extension system, so new tasks can be created via composition of existing tasks.</li>
<li>A Maven plugin.</li>
</ul>

<p>I'd love to receive feedback on your experiences with Lightest. Drop by the project page at <a href="http://code.google.com/p/lightest">http://code.google.com/p/lightest</a> or file a bug or enhancement request. Also, I hope to get a new website up at <a href="http://stressfreetesting.com">http://stressfreetesting.com</a>, where Lightest will be a spotlight techonology - there should be more information available there too. Enjoy!</p>

<p><em>- Haw-Bin Chai (hbchai @t gmail d0t com)</em></p>

<br />
<br />
<br />
<br />

</body>
</html>